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Abstract 

In this study, three attributes of faculty competence-- teaching, research* 
and service—were each measured by three different instruments— department 
chair evaluation, self-evaluation, and one other measure (student evaluations, 
number of publications, and number of committees). The data were analyzed 
through a mul ti trait-multi method matrix to ascertain convergent and divergent 
validity. The results indicate that the evidence for validity is relatively 
weak; it was concluded that the results from these measures should be 
interpreted with considerable caution. 
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Convergent and Discriminant Validation of Measures of Faculty 

Competence by the Mul ti trai t-Mul ti method Matrix 

The study of measures of faculty competence has received considerable 
attention in the higher education 1 i te rat lire, particularly during the past 
two decades, with the strengths and limitations of numerous criteria of 
performance being discussed (Miller, 1979). From a national survey of department 
chairmen, Centra (1979) reported that the four major sources of info rmation 
on faculty competence were chairman evaluations, col league opinions , student 
ratings , and committee deliberations. While some publ ished studies have 
reported correlations among measures from these sources (Blackburn & Clark , 
1975) , the generalization of such results requi res evidence of validi ty ; namely, 
that the various sources of information can consistently distinguish among 
di fferent aspects of faculty competence and can agree when addressing the same 
aspect. 

_ In their attempt to devel op a methodological approach to val i di ty which 
would lead to a more sophisticated unders tanding of psychological traits , 
Campbell and Fiske ( 1959) , in a now classic article, proposed the concepts of 
convergent and discriminant validity. They described the mul ti trai t-mul timethod 
matrix (MTMMM) as an efficient tool for displaying correl ations , and developed 
cri teria for eval uating the MTMMM in terms of the proposed validity concepts. 
To apply this technique , several traits are each measured by several methods, 
with the intent of showing the distinctiveness of each trait, regardless of 
the method of measurement used. Convergent validity is evidenced when di fferent 
methods of measuring the same trait show high correl ations , while discriminant 
validity is evidenced when di f fe rent traits measured by the same method produce 
low correlations. 

The MTMMM has been widely used in the literature, where it has made 
notabl e contributions toward the investigation of human traits (Bajtelsmit, 
1979; Lawler, 1967) . Recently, the technique was used to investigate factors 
of teacher effectiveness, as measured by student ratings and instructor self- 
evaluations (Marsh, 1982) . 

When employi ng the MTMMM, the researcher is fi rs t confronted with the 
ques ti on of what trai ts are to be assessed. Numerous traits have been proposed 
as components of facul ty competence ; e.g., Centra (1979) reported ten, 
Mi ller ( 1974) reported nine, and Marsh (1982) reported nine. Since considerable 
overlap existed among many of these , it appeared appropriate to conceptual ize 
them into the traditional "three-legged stool' 1 of teaching, research, and 
service. It thus became the purpose of this study to investigate the convergent 
and discriminant validity of several common measures of these three tradi tional 
attributes using the logical analysis of the MTMMM approach. 

Method 



Subjects 



The data for this research were obtained from eight departments in a 
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midwestern university (Hansen, 1979). Within eaeh department, a random sample 
of faculty members was drawn proportional to the department size, arid the__ 
resulting 111 faculty members were invited to. participate in the study,. Qf 
the 73 who agreed to do so , complete information was obtained from 57 (4 females 
and 53 males). The ^e were 22 ful 1 professors 22 associate professors , and 13 
assistant professors, all of whom held the doctorate. 

Procedure 



The MTMMM approach requires that the same multiple methods be used to 

evaluate each of the traits. For this study, three sources of evidence were 
sought^ First, the appropriate department chair rated, on a scale of 1-7, the 
professors performance in the areas of teaching, research, and service for 
two time periods ;.. namely; for that .current academi c year and for the compos i te 
of al 1 prior yea rs Second, each professor rated, on a scale of 1-7, his or 
her own performance in the same three areas for both time periods. Finally, 
one other measure of performance was collected for each of the ^three traits. 
Current '.teaching performance was measured by the mean student response to a 
question concerning the professor's overall teaching ability. No measure of 
previous teaching ability was available. Using data from a faculty questionnaire 
covering both time periods, research performance was measured by total publi- 
cations (including books, textbooks, journal articles, and. other publications), 
arid service performance was measured by the total number of department, college, 

and university committees on which the faculty member served. Since these ; 

last three measures did not employ the same method, the analysis of the MTMMM 
was appropriately modified. 

Resul ts 



Pearson product moment correlation coefficients were computed among all 
of the variables and cast into a MTMMM (see Table 1). The parenthesized values 
in the main diagonal represent reliability estimates > and the underlined values 
represent validity coefficients. Each area of the matrix will be explained 
below. 

The reliability values were obtained by correlating the results represent- 
ing _the two time periods. It would have been preferable to have obtained these 
coefficients by gathering data at two different times. However, since both 
ratings were, in fact, obtained at the same time, the first ratings represent 
the assessor's memory of how the faculty member would have been rated previously. 
This approach assumes a perfect memory on the part of the rater, a highly 
questionable assumption (Kenny, 1979, p. 233). _ Although these were the most 
stable reliability estimates that were available for this study, the resulting 
values should be interpreted with caution. 



As expected, the reliability coefficients in the MTMMM tended to be the 

largest values within the matrix; indeed, they appeared sufficiently large 
to warrant further examination of the convergent and divergent validity of 
the measures. 

Convergent validity was investigated by inspecting the validity diagonals; 
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namely, those correlations between different methods of measuring the same 
trait (the monotrai t-neteromethod correlations). Of the nine validity 
correlations, only five exceeded the critical value of .27 (frr a sample size 
of _57.at the .05 significance level). Al though the evidence for convergent 
validity is somewhat weaker than might be desired, it is not completely 
unexpected. Marsh (1982, p. 275) reported a median correlation of only .30 
between instructor self-ratings and ratings by %hei r students. When the data 
of Blackburn arid Clark ( 1975) are conceived of as a MTMMM, the validity 
diagonals show a median value of .45 with a range from .10 to .63. The results 
o f the present study are not inconsistent with those values. 

Discriminant validity was investigated with three criteria. First, two 
methods of measuring the same trait should correlate higher than two measures 
having neither trait nor method in common . That is v a value in a validity 
diagonal should be higher than the other. values in its column and row in the 
heterot^ai t-heteromethod triangles. Table 1 shows. that the three validity 
values for Teaching (.20, .36, arid .34) were each larger than the values in 
the corresponding rows and columns within the dashed triangles . The variable 
Research showed similar results, but Service exhibited weaker evidence. 

The second criterion states that two methods of assessing a particular 
variable should show a higher correlation than should the same method when 
applied to that particular variable and some other variable. That is, the 
validity values in a particular column (or row) should be higher than the 
heterotrai t-monomethod values in the same column [or row),, if method effects 
are to be eliminated. From the data in Table 1, there was an indication of 
strong method effects. For example, when the validity values of .20 and .36 
for Teaching were compared with the values of .23 and .56 in the same column, 
a method effpct for department chair ratings could not be ruled out. While 
the validity coefficients , on the average, tended to be higher than the 
corresponding heterotrai t-monomethod values there were still enough exceptions 
to give cause for concern. 

The third criterion suggests that a similar pattern of trait inter-relation- 
ships should be shown regardless of the method used. Thus* a similar pattern 
of correlations should be found within each of the triangles. However, from 
the data in Table 1, it was difficult to detect any consistent pattern within 
the triangles. 

Discussion 

Applying the MTMMM approach to the measures of faculty competence in this 
study led to somewhat ambiguous results, because the required conditions were 
not fully met for each criterion. Some writers have suggested that, jn certain 
eases, path analysis can provide less ambiguous results than the Campbel 1-Fiske 
criteria (Schmitt, 1978). _ However, si nee path analysis was primarily developed 
to handle situations involving large numbers of traits and methods, the 
Campbel 1-Fiske analysis. appeared more appropriate for the present study, 
considering the few traits arid variables therein. 

There are several factors that should be considered in interpreting the 
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findings of this study. First, the sample size was small. No doubt,, more 
stable results would have appeared with a larger number of subjects, but 
research using confidential type data often must be restricted to relatively 
small sample sizes. Second, certain properties of the instruments might be 
questioned. Were the chair evaluations given for this study the same that 
the chair actually used when making official reports? Was the investigator- 
made student evaluation instrument as reliable and valid as instruments used 
elsewhere? Was the number of publications and the number of committees 
synonomous, respectively, with the quality of research and service? Again, 
the nature of the study imposed certain limitations on the types of 
operational definitions that could be used for the constructs. Third, the 
three traits may each have been a composite of other trai ts . When Marsh 
(1982) factor analyzed student rating scales, the results indicated several 
distinct factors, and he suggested, that assuming student ratings to be 
uni-dimensional may be one factor leading to ambiguous results. 

While the above limitations should be considered in interpreting the 

results, the fact that confidential data of this type are difficult to obtain 
suggests both that these tentative conclusions may be of value to other 
researchers, and that further investigations are warranted. 

In summary, the MTMMM analysis indicated that the three hypothesized 
components of faculty competence could be distinguished one from another, but 
that the evidence for their convergent and discriminant val idi ty was relatively 
weak. Therefore, while having some value, data purporting to measure such 
trai ts shoul d be interpreted with considerable caution. 
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Table 1 

Multi trait - Muiti method Matrix of Measures of Faculty Competence (N = 57) 



Dept. Chair 
Traits Tl Rl Si 

1. Dept. Chair 

Teaching (Tl) 
Research (Rl) 
Service (SI) 

2. Self 

Teaching (T2) 
Research (R2) 
Service (S2) 

3. Other 

Teaching (T3) 
Research (R3) 
Service (S3) 



Note. Parentheses are used to identify the values in the reliability 
diagonal. (The reliability of the Teaching variable was not available for 
these data.) Underlines are used to identify the values in the validity 
diagonals. Solid line triangles enclose the heterotrai t-monomethod values. 
Broken line triangles enclose the heterotrai t hete ro method values. The inter- 
correlations among the Other methods are not enclosed by solid lines because 
they are not mono-method. Decimal points are omitted. 
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